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Results of the Survey of RP Group Members: 

An Element in Strategic Planning for the Research & Planning Unit 

Willard Horn' 

California Community Colleges Chancellor’s Office 

Introduction 

When an organization has a research function, the organization often delegates 
that function to a subunit. That subunit, often called the “research unit,” will need to 
adapt to its internal and external environments in order to function productively and 
survive. As a means of adapting to changing environments, such research imits will 
periodically collect data about their environments in order to plan adaptive strategies. 
Although this paper focuses upon a stakeholder survey for a research unit, all types of 
organizations or units undertake such efforts for the same reason (Bryson, 1988, pp.l03- 
104; Wildavsky, 1980). 

Naturally, information about how stakeholders perceive a research unit’s 
performance plays an important role in understanding adaptive needs. Another very 
useful type of information is the description of stakeholders’ research needs and 
perceptions of how the unit can help meet those needs. Both types of information can 
result from a survey of various stakeholders in this adaptive process (which we labeled as 
a strategic planning process). This article covers just one part of the overall strategic 
planning process that the Research & Planning Unit at the Chancellor’s Office for the 
California Community Colleges (henceforth the RP Unit) has begun so that it can adapt 
itself to changing times. The part covered here is the survey of one stakeholder group, 
the Research and Planning Group for California Community Colleges. Although 
resources will not allow us to discuss input from other external stakeholders and internal 
stakeholders (parties within the Chancellor’s Office), we did survey other groups in this 
strategic planning effort. 

As an instrument of our environmental scanning process, this stakeholder survey 
focused on some general propositions that relate to our strategic planning. We list these 
propositions as follows: 

1. Five factors (credibility, responsiveness, analytical capability, clarity of 
communication, and relevance) contribute to overall satisfaction with RP Unit 
performance. 

2. Simple performance ratings of the RP Unit will gain valuable contextual meaning 
when the respondent is asked to explain those ratings with words (in response to a 
follow-up question in an open-ended format). 

3. Institutional researchers at the commimity colleges perceive many different 
system-level research needs. 
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4. Institutional researchers at the community colleges perceive specific areas of 

research need that the RP Unit can or should address (but these areas will differ 

from what they perceive to be system-level research needs). 

Proposition #1 will help the RP Unit understand how its perceived performance 
relates to five dimensions of service delivery that theoretically apply to service providers 
(both in the public and private sectors) that provide research. In this vein, analytical 
capability distinguishes this satisfaction model from those models that researchers use for 
other types of output, such as human services (e.g., counseling, health care, and 
instruction). 

Proposition #2, if true, will help the RP Unit understand how respondents derive 
their ratings. Researchers have recommended following up close-ended questions with 
an open-ended question for precisely this purpose (Groves, Fultz, & Martin, 1992). This 
information will contextualize the ratings data by revealing some of the thought processes 
involved. To a great extent, this narrative data informs the RP Unit about the diverse 
paths which stakeholders take to judge the RP Unit’s performance on the six measured 
dimensions. Collectively, the narrative data will help the RP Unit to understand how it 
can enhance or maintain stakeholder satisfaction with its work. 

Proposition #3 explores the scope and depth of perceived system-level research 
needs. To a large extent, the RP Unit expects to find topics that it has not recognized in 
its recent operations. From a strategic planning perspective, this proposition deals with 
the basic demand components for system-level research. In some cases, the RP Unit can 
use this information to provide indirect assistance. For example, the Unit could advocate 
resources for local efforts, facilitate the use of research consortiums or “think tanks” to 
conduct broad-ranging research, or support data collection systems that help meet 
emerging analytical needs. 

Proposition #4 explores the scope and depth of local research needs that 
institutional researchers believe can benefit from the RP Unit’s direct efforts. Whereas 
proposition #3 addresses local perception of system-level needs (that the RP unit may or 
may not be expected to address), proposition #4 helps the unit learn about areas for which 
institutional researchers would like direct assistance from the RP Unit. Subsequently, 
such information will presumably help the unit, resources permitting, to direct its effort to 
areas that have relevance to local needs. 

Data Collection 

Unit staff created a brief questionnaire for implementation as an e-mail survey. 
The questionnaire had six items to be rated and three open-ended questions. (See the 
appendix for a copy of the questionnaire.) The rating items provide us with a basic 
measure of respondents’ overall satisfaction with the RP Unit and with measures of five 
factors that theoretically should relate to customer satisfaction among research users. 
This approach parallels the modeling done in prior work for private sector service 
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providers (Zeithaml, Parasuraman, & Berry, 1990; and Parasuraman & Zeithaml, 2002). 
From a planning perspective, we needed to know how satisfied RP Group members were 
with RP Unit performance, and we simultaneously wanted to test five commonly 
perceived performance factors in terms of their relationship with overall satisfaction 
among these stakeholders. (Note: In the rest of this article, we shall display the survey 
rating items in italics to distinguish them for the reader.) 

We used the RP listserve to distribute the e-mail survey and a cover letter to 
obtain an economical coverage of the target population (the RP Group members). The 
survey occurred during the summer of 2002. We received 39 survey responses. Because 
the listserve membership is a fluctuating and uncertain number, an exact response rate 
was not calculated. The listserve probably had a population of about 200 individuals 
which would translate into a response rate of 19.5 percent. This, of course, should 
prompt users of the response data to think about nonresponse bias before drawing 
conclusions. 

In addition, the RP listserve has some shortcomings as a sampling frame, 
especially in terms of sample undercoverage and overcoverage. Some listserve members 
do not have membership in the RP Group, and they would not be in the target population 
for this survey (overcoverage). On the other hand, some colleges have RP Group 
members who are not listserve members (under coverage). 

The numeric ratings came from 30 to 34 respondents, depending upon which of 
the six rating dimensions is considered. Five respondents to the survey skipped the 
ratings altogether or marked the “Don’t Know” option for all of the ratings. 

Analysis 

The histograms in Figures 1 and 2 indicate “end-piling” on the high side of the 
scale. That is, few ratings occurred on the low end of the scale. The dimension of 
relevance had the greatest variability in these ratings. This seems logical considering that 
each respondent probably had different frames of reference with which to use in rating 
relevance. Apparently, campus needs and research office concerns vary widely across 
the state. 
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Figure 1: Distributions of Ratings for Credibility, Responsiveness, and Analytical 
Capability 
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Figure 2: Distributions of Ratings for Clarity, Relevance, and Overall Satisfaction 
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Table 1 presents the summary statistics for the six ratings. Relevance had the lowest 
mean rating, and analytical capability had the highest mean rating. These results are 
understandable in that the RP Unit may demonstrate very sound technical competence but 
on issues that some individuals in the field feel have low relevance to them. The fact that 
the mean ratings for credibility^ responsiveness, clarity, and overall satisfaction are high 
on this 10-point scale indicates a positive overall evaluation from the respondents. The 
variation between the dimension ratings indicates the validity of the ratings in that 
individuals exhibited discrimination in their responses. 

Table 1. Summary Statistics for the Six Ratings 



Variable 


Observations 


Mean 


Std Dev 


Min 


Max 


Credibility 


30 


7.83 


1.18 


5 


9 


Responsiveness 


30 


7.67 


1.24 


5 


9 


Analytic Capability 


34 


8.03 


0.97 


6 


9 


Clarity of Communication 


34 


7.18 


1.31 


4 


9 


Relevance 


33 


6.84 


1.75 


3 


9 


Overall Satisfaction 


34 


7.59 


1.10 


6 


9 



The relatively low mean rating for clarity also may make sense when technical material 
(such as statistical modeling) is part of the basis for evaluating the RP Unit. Highly 
technical material presents an additional challenge for the RP Unit because it cannot 
assume that all of the field researchers have the same backgrounds or interests as the RP 
Unit has. Of course, writing and speaking ability may also play a part in this rating for 
clarity. It is unclear how much the RP Unit’s writing or speaking ability has influenced 
the rating of clarity. 

Proposition #1 explores the relationship of five performance factors to overall 
satisfaction. Although a statistical analysis of the ratings cannot prove this proposed 
relationship, the finding of no statistical correlation would tend to disprove this proposed 
causal link. Consequently, staff computed Pearson correlation coefficients and output 
them in the form of a correlation matrix. 

This correlation matrix, shown in Table 2, seems to support some expectations. 
Responsiveness had the strongest correlation with overall satisfaction. This linkage 
agrees with what leading researchers in customer satisfaction have found (Zeithaml, 
Parasurman, & Berry, 1990). Analytical capability had very little correlation (or 
practically none) with either clarity or relevance, but it had moderate correlation with 
credibility and overall satisfaction. Clarity had a strong correlation with relevance and 
overall satisfaction. 

Some caution is necessary when comparing the Pearson correlation coefficients in Table 
2. The analyst should test for a statistically significant difference in these coefficients 
before interpreting the data. This is necessary for the analyst to avoid mistakenly using a 
difference caused by sampling error as a true difference among correlations in the target 
population. 
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In statistically comparing the correlations of the five factors with overall satisfaction, 
only the factors of analytical capability (r = 0.4366) and responsiveness (r = 0.7408) 
really differ at the .05 significance level. This testing used the Fisher’s Z transformation 
for correlations and the normal curve table (Guilford & Fruchter, 1978, pp. 163-1 64; 
Cohen & Cohen, 1983, pp. 53-57). 

Table 2. Correlation Matrix of the Six Ratings 



Variable 


Cred 


Resp 


Anal 


Clarity 


Relevance 


Overall 


Credibility 


1.00 












Responsiveness 


0.54* 


1.00 










Analytic Capability 


0.57* 


0.19 


1.00 








Clarity of Communication 


0.47* 


0.47* 


0.11 


1.00 






Relevance 


0.32 


0.02 


0.44 


0.00 


1.00 




Overall Satisfaction 


0.56* 


0.74* 


0.44* 


0.68* 


0.66* 


1.00 



* p < -05 



Question #2 (the follow-up open-ended question to question #1) provided valuable 
information for understanding the numeric ratings staff received. The narrative clearly 
explained the respondents’ thoughts that influenced their ratings (or non-ratings on 
occasion). In general, the narrative responses to question #2 validate the ratings data, 
especially because they indicate both the level of cognitive effort put forth by the 
respondents and an understanding of the ratings. 

For example, one respondent wrote the following comments for question #2: 

I think that the technical competencies of the staff are outstanding, however, the field is 
not, as a rule, up to the same level as the unit, therefore the unit needs to do its best to 
ensure that the field understands what the unit has done and why. Merely quoting a 
statistic alienates the field from the good work that the unit is trying to perform. From a 
relevance standpoint, the state is often focused on policy decisions, which may have little 
to do with the operational decisions that take place on college campuses. The state has 
the competencies to assist colleges with relevant research, e.g., benchmarking, student 
outcomes, etc. It would be a great service to us all to have these studies. At one time, 
student outcomes for the system were reported under the "Selected Performance 
Measures" report. If that one report was resurrected, I think folks would be very happy. 

This individual had given ratings of 9 for analytical capability, 1 for credibility, and 7 for 
responsiveness. However, the other ratings were 4 for clarity of communication-, 4 for 
relevance-, and 6 for overall satisfaction. (The questionnaire’s rating format had an upper 
limit of 9 and a lower limit of zero.) The comments not only support the variation in 
ratings, but they also explain the factors and reasoning used by the respondent in the 
rating process. 

Another example demonstrates this point with the following comments from a different 
respondent: 
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I know there has been significant change in both staff and leadership in the past few 
years, so I'm not very familiar with what this staff has accomplished in the past few 
years. However, I do see some very very good research things coming from the 
Chancellor's office. For example, the study of transfers and transfer effectiveness was 
great. (I hope you get which report I'm referring to). The web site has been developing 
very rapidly recently with easy access to lots of useful data. Some reports presented at 
the RP Group conference were excellent. 

The above respondent gave ratings of 8 for credibility, no rating for responsiveness; 8 for 
analytical capability; 7 for clarity; 5 for relevance; and 7 for overall satisfaction. 
Because the respondent’s contact seemed to exclude any requests of the RP Unit there 
was no rating for responsiveness, and the ratings for clarity, analytical capability, and 
overall satisfaction were high despite a relatively low rating for relevance. 

While the narrative responses to question #2 often demonstrated the validity of the 
ratings (as well as explaining them), they also indicated the potential for attribution error. 
That is, the respondent may attribute work to the RP Unit that it did not perform. The 
following response is an example; 

I am new to the field of institutional research (vintage Nov'Ol). At this point, it is difficult 
for me to distinguish between the Chancellor's Office, as a whole, and the Research & 
Planning Unit in particular. Each of the several inquiries I have placed to R&P Unit staff 
have been handled in a responsive manner. 

An example where attribution error did occur is the response to question #2 from a 
different person: 

Any time I have needed help, “Employee X” [an MIS official] has found the solution, or 
handed it over to capable staff who found the solution. 

The above respondent gave ratings of 9 for responsiveness and 8 for overall satisfaction. 

In terms of the five returned surveys that lacked any responses to the rating items, the 
follow-up question provided some information about the nonresponse for ratings. For 
example, one text response to question #2 was the following: 

I have a hard time rating my own credibility, responsiveness, etc. let alone trying to rate 
yours. We are a fairly new office of one technician, period. You are my only standard. 
Everyone I've had contact with has been quick to respond, clear in their communication, 
and more competent than I. For that, I am thankful. However I am unable to measure the 
above dimensions based on my naive and limited experience. 

The responses to the other open-ended questions provided quite extensive narrative data. 
A summary tabulation of the kinds of research topics/areas appears in Table 3. This 
summary table indicates both the divergence and overlap (or consistency in viewpoints) 
in the opinions of the respondents. The “other” category has a relatively large count 
because so many responses had unique categories. The number of unique responses 
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captured in the “other” category indicates that a number of research topics may have very 
narrow bases of support or interest. From a research administrator’s viewpoint, these 
unique responses, though possibly important to the system, lack anything near a 
consensus of opinion among this set of respondents. 

Table 3. Tabulation of Coded Responses to Open-ended Items 



Topic of Interest 


System Need 


Local Need 


Student Outcomes 


12 


5 


Assessment 


8 


2 


Basic Skills 


8 


2 


Transfer 


6 


2 


Accreditation/Assessment 


5 


2 


Enrollment Trends 


4 


2 


Workforce Development 


4 


1 


Institutional Effectiveness 


3 


0 


Partnership for Excellence 


3 


1 


Matriculation 


3 


2 


Student Tracking 


3 


2 


Program Review 


2 


0 


Environmental Scan 


2 


0 


Data Sharing 


2 


3 


Distance Education/Technology 


2 


1 


Data Integrity 


2 


1 


Peer Group 


2 


1 


Quantitative Models for Colleges 


0 


2 


Other 


17 


20 



Note: Table includes multiple responses from some respondents. 



While unique topics may lack much broad support, they may embody valuable issues to 
consider for a system-level research agenda. For example, two respondents gave the 
following unique responses: 

Development of the qualitative and political analytic techniques for institutional analysis. 
[Respondent A] 

Is the shift to measurement of student learning outcomes contributing to teaching and 
student learning? ...Preparation and recruitment of ethnic groups who have somewhat 
lower-than-average participation rates in college attendance. Assessment of extent to 
which 2-year colleges are preparing students for successful achievement at the 4-year 
institutions. [Respondent B] 

Table 4 tabulates the unique items in the “other” category. A number of these items 
(such as economic development, ESL, and outreach) have obvious importance to policy 
makers. Thus, there is some risk in ignoring the unique research topics to focus upon the 
topics that garnered more frequent reporting by respondents. 
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Table 4. Tabulation of Unique Responses to Open-ended Items 



Topic of Interest 


System Need 


Local Need 


Effectiveness Report 




1 


Qualitative Methods 




1 


Classroom Research 


1 




MIS/Data Element Dictionary Training 




1 


Data Costs 


1 


1 


Simultaneous Degree Programs 


1 




ESL 


1 




SPSS Training 




1 


Cut Score Validation 


1 


1 


Asynchronous Environments 


1 




Counseling 




1 


Funding 


1 




Measuring Student Outcomes 


1 




Outreach 


1 




Accountability 


1 


1 


Missing but Needed Research 




1 


Toolbox for Researchers 




1 


Clearinghouse for Information 




1 


Student Satisfaction 


1 


1 


Economic Development 


1 




Report Standards 




1 


Best Practices 




1 


Library 




1 


Data Mining 




1 


Persistence 




1 


Special Populations 


1 




Teacher Training 


1 




Non-English Speaking Populations 


1 




Articulation (Math and English) 


1 




Regional Research 




1 


Compressed Term 


1 


1 


IR Director Training 




1 



Note: Table includes multiple responses from some respondents. 



A comparison of responses to questions 3 and 4, in Table 4, also informs us about the gap 
between perceived system-level research needs and research efforts that local researchers 
feel the RP Unit can do for or with them. The general trend in the table is that although a 
topic may have system-level importance to the local researcher, only a small proportion 
of these researchers seem to find that topic suitable for direct assistance from the RP 
Unit. 
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Discussion 



The survey results supported the four propositions stated at the beginning of this 
article. Credibility, responsiveness, analytical capability, clarity of communication, and 
relevance all correlated positively with overall satisfaction. Despite the modest sample 
size (34 or less for each of the ratings), each correlation was statistically significant well 
beyond the standard of the .05 (or the .01) significance level. We can thus infer that the 
correlations were unlikely to have resulted from sampling error. Given this finding, the 
Unit probably should continue to pay attention to each of these five dimensions in order 
to maintain a high level of stakeholder satisfaction. 

One result may surprise some observers. Many people believe that researchers 
value, and desire to use, the services of a provider with high capability for analyses. On 
the surface, analytical capability’s relatively low correlation of 0.44 with overall 
satisfaction seems counterintuitive, compared to the 0.74 correlation of responsiveness 
with overall satisfaction. But an interesting dynamic may operate here. One factor may 
be the attitude that much of the needed research is basic in nature. That is, basic 
descriptive statistics will fulfill many of the local demands; sophisticated elaborative 
models are superfluous. Furthermore, local researchers may prefer studies or methods 
that they can easily convey to their administrative hierarchies. So complex studies may 
get a chilly reception before busy administrators (to whom local researchers must 
answer). 

The correlation matrix in Table 2 may help explain the status of analytical 
capability. This factor correlated either lowly or not at all with responsiveness, clarity, 
and relevance. In contrast, responsiveness, clarity, and relevance, all correlate with each 
other at least at a moderate level. But analytical capability only correlated with 
credibility among the four other factors. Staff calculated that analytical capability’s 
correlation with credibility has a statistically significant difference (at the .05 level) from 
analytical capability ’s correlation with clarity, relevance, and responsiveness. 

So analytical capability may largely contribute to overall satisfaction via its link 
to credibility. Analytical capability somehow differs from credibility, clarity, relevance, 
and responsiveness in its linkage with overall satisfaction. This could imply that 
analytical capability really matters when stakeholders, like researchers, weigh credibility 
heavily in their judgment of overall satisfaction for a research provider. 

Question #2 (the open-ended follow-up question to the ratings) provided ample 
explanation and support for the validity of the ratings. The implication for other survey 
planners is that such follow-up questions can help the researcher to check the validity of 
numeric ratings. At the same time, the follow-up open-ended question can provide 
invaluable insights about the thought processes of respondents. Of course, the nature of 
the survey’s target population will seriously affect the level of success the survey 
researcher will have. In this study, the RP Group was an ideal population. Many of the 
group’s members have (1) a high interest in and commitment to the survey and the 



11 



05/05/2003 



administrative use of the responses; (2) a graduate education; and (3) experience in 
conducting their own surveys. 

The institutional researchers at the community colleges have many diverse 
research needs. In fact, the data indicate that there is little, if any, consensus for a 
particular research topic. The recognized need for research includes many topics that 
seem to have only pockets of support. From a planning perspective, the RP Unit has a 
good situation and a dilemma at the same time here. The wide array of needed research 
implies that the Unit could have much flexibility (or choice) in serving those needs. Such 
flexibility is useful in the sense that the Unit could achieve efficiency in its output by 
sequencing and selecting research projects that fit its schedule and capacity. On the other 
hand, the existence of a consensus research topic would improve research efficiency. 
That is, the unit could meet the needs of many different stakeholders across the state by 
completing one widely supported and widely needed project. 

The set of local research needs that institutional researchers believe are most 
amenable to RP Unit assistance differs from the set of research needs that this group 
conceptualizes as system-level. Research needs that involve data (e.g., data sharing) 
naturally fit the set of local research needs because much analysis requires data that 
individual college researchers have difficulty in obtaining. For example, research on 
transfers, student tracking, and workforce development uses data captured and 
maintained by entities like the Chancellor’s Office of the CCC, which have state-level 
authority and capacity for such functions involving multiple community college districts. 
Other logical areas of RP Unit contribution to local research involve multiple district 
analyses, such as enrollment trends, benchmarking, peer grouping, environmental 
scanning, and PFE analyses. To a certain extent, these local needs are immediate, in 
contrast to the scope of question #3, which specified a five-year planning horizon for 
research needs (while question #4 specified no timeframe). From a planning perspective, 
however, the main point is to realize that recognition of a system-level research need is 
different from locally specified need for research by the RP Unit. 

Of course, this identification of various topics as tasks for the RP Unit could 
dramatically change with a number of circumstances. If local budgets for research 
activities were to shrink substantially, community colleges may expect the Chancellor’s 
Office to assume some of this unfunded workload. To the extent that multi-district 
research consortiums successfully develop across the state, notwithstanding budget 
issues, the institutional researchers may recognize less need for research work from the 
RP Unit. In a different scenario, it may serve local interests to expand the declared needs 
for local research (to be done by local researchers) in order to justify budget requests (or 
to defend existing budgets). In areas where multiple districts may conflict over a topic, 
the RP Unit may play a vital role as a neutral, impartial party to produce equitable, 
mutually acceptable research. 

The above paragraphs have addressed the research-related propositions that the 
survey helped explore. An important administrative purpose for the survey, in contrast to 
research purposes, is measurement of one of the many indicators for performance by the 
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RP Unit. The next few paragraphs discuss the survey results in terms of RP Unit 
performance. 

Table 1 in the Analysis section captures the gist of the ratings. Without any 
benchmarks for these ratings however, the numbers lack an evaluative context. 
Fortunately, the text responses to question #2 help provide this evaluative context in 
detail. The positive overall evaluation in the ratings receives corroboration with text 
comments like the following: 

• Excellent team and excellent work. 

• I know there has been significant change in both staff and leadership in the past 
few years, so I'm not very familiar with what this staff has accomplished in the 
past few years. However, I do see some very very good research things coming 
from the Chancellor's office. For example, the study of transfers and transfer 
effectiveness was great. (I hope you get which report I'm referring to). The web 
site has been developing very rapidly recently with easy access to lots of useful 
data. Some reports presented at the RP Group conference were excellent. 

• All my experience with the R&P Unit staff has been positive and productive. 
Timely responses, clear communication. 

• Whether asking a question of unit staff by phone or attending a presentation given 
by R&P staff, I have always been given complete, clear, accurate information in a 
professional and personable marmer. I appreciate knowing that they are available 
as a resource. 

• My ratings are based upon my experiences with your office as relative newcomer 
to the RP group — the past two years. I have always received excellent response 
when contacting individuals at your office for information or data requests, and I 
have done so several times. I have also attended several of the workshops 
presented at various meetings — CAIR, RP — and found the presentations to be 
valuable for my work. 

In particular, respondents recognized the laudable intentions of the RP Unit and its 
technical competence. 

• I am new to research, less than 1 year, and most of my experience with the 
CCCCO has been in relation to MIS data. I am excited to see the continual 
improvement over the last year and the desire to assist the colleges. Thank you 
for providing the services that you do. I have learned quite a lot about educational 
research from the CCCCO. 

• ...[the director’s] initiatives and efforts are appreciated; I think he has a good 
sense of where his office needs to go. Also, I see an improvement on the 
expertise and responsiveness of the office... 

In balance, however, respondents noted shortcomings and areas of desired improvement. 
The following comments make these kinds of points: 
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• Communications aren't always clear. There seem to be too few trying to do too 
much for too many. 

• . . .1 don't see much of what is produced. Am I not paying attention or is the work 
not being "advertised" on the RP listserv? Or am I missing direct announcements? 
When I went and looked, my only comment is: if I can't use the results, I'm not 
much interested. Sorry to be obnoxious but I've got several specific problems that 
I'm concerned about, the foremost currently is enrollment, and not a lot of time or 
interest in academic style research. 

• I believe that the Research & Planning unit is too dominated by mathematical 
models that don't fit credibly into the process of planning. Within this limitation 
however the materials developed and provided are first rate, as is the timeliness of 
the interventions. Similarly, while the technical competence of the RP group in 
the quantitative/statistical component is very good, the absence of a qualitative 
dimension and the subordination of research to predetermined policies limit the 
possible relevance to actual practices. I have the distinct feeling that the RP unit 
does not tackle some of the difficult political problems that determine whether 
and especially how research gets used on the campuses. The tail of existing 
policies wags the research dog. 

• Overall, the unit does a good job. Sometimes the communication becomes a little 
too technocratic and stiff, but all-in-all, the support is quite helpful. Because so 
much of the Research and Planning unit's work is highly analytic, it is sometimes 
difficult to convey results without extensive reporting of fairly sophisticated 
statistical procedures. 

• I think the unit does fine work, but its mission, as I understand it, is not to support 
the “field”. In recent papers the staff has made major contributions to the 
dialogue of research work in our system. 

• In general, this unit appears to selectively provide data and reports that are either 
for legislative, administrative or compliance with a funding situation. With the 
large amount of data available through the MIS database it would seem that your 
office could broaden its horizons to evaluate and report on a number of topics. It 
would appear from my standpoint, that the research office goes on research 
fishing expeditions and only keeps and publishes information on the attractive 
fish. So what are the benefits to the field from feel-good research? 

• Having a difficult time remembering research that was relevant to our needs, 
except the environmental scanning piece. 

• Most of all research and planning needs marketing in the CC system. The 
discussions among CEO’s should focus research priorities and the RP Group 
should play a bigger role in Vocational Education research. We don’t hear much 
about CCCCO research. It seems quiet. We never conduct any research studies in 
conjunction with you guys. We’d like to, but have never come together through 
the RP Group to coalesce. The best information has been on Transfer students. 

Not only do the text responses provide an evaluative context for the ratings, they 
also convey some means of enhancing performance. Aside from the service delivery 
dimensions (such as relevance to the field and clarity — esoteric level of communication), 
the following suggestions, among others, warrant consideration in RP Unit plans: 
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1 . Increased collaborative research work with members of the field 

2. Broadening the guidance of research to include other pools of expertise. 

3. Marketing of the research function 

4. Increased attention to vocational education 

5. Increased effort to support the field. 

Some suggestions for improvement will be difficult to enact. Many important 
analyses cannot be done simply because the Unit lacks resources, and existing areas of 
expertise and jurisdiction (some topics are the “turf’ of other entities) create limits to the 
scope of Unit work. 

The dramatic shift in the budgetary environment certainly makes these findings 
less applicable in the short-term horizon. Whether these findings fail to reflect the long- 
term horizon will of course depend upon any overall shift in the institutional research 
environment that the current state budget crisis will have caused. Furthermore, staff 
vacancies that have resulted from budget reductions in the Chancellor’s Office may make 
it difficult for the RP Unit to maintain or to improve the performance that respondents 
reported in this survey. 

Likely staff changes among the institutional researchers in the community 
colleges, whether precipitated by budget reductions or not, will also affect any future 
survey results as well. New researchers will eventually assume the roles that current 
researchers hold, if future budgets permit, and new functions and tasks at the local level 
may accompany such periods of transition. The RP Group may eventually have members 
with very different expectations and needs than the members of today. Although it may 
be difficult to compare any future survey results of RP Unit performance with the current 
survey results, the use of a future replication of this survey would still be worthwhile. In 
a few years, the RP Unit will again need to monitor its environment to update its plans 
and to adapt to the shifting environment. 

Limitations 

In reporting research results, authors should help readers recognize any 
qualifications that should apply to a study. There are some qualifications worth noting 
for this analysis. 

1. The low response rate and the sample frame’s coverage issues limit the extent to 
which we can generalize these findings (Agresti & Finlay, 1997, pp. 23-24). 

2. Additional factors could contribute to stakeholder satisfaction, but this initial 
investigation could only investigate five of them at this time. The potential 
existence of untested factors could affect how a strategic plan would handle the 
issue of stakeholder satisfaction. 

3. The evidence of correlations linking the five performance factors to overall 
satisfaction may really be a case of spurious correlation. That is, the factors may 
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link statistically to overall satisfaction but they may not necessarily link causally 
to overall satisfaction (Agresti & Finlay, 1997, pp. 357-365). 

4. In a rigorous study of overall satisfaction, the researcher would use a multi-item 
scale for the construct of overall satisfaction to achieve added reliability and 
validity (Eagly & Chaiken, 1993, pp. 80-81). 

Conclusions 

This survey’s ratings provide the RP Unit a glimpse of how one set of 
stakeholders views its performance. The survey’s open-ended questions provide the RP 
Unit detailed information about specific concerns and needs that institutional researchers 
may have. The survey results provide support for the continued use of close-ended 
questions in concert with open-ended questions to paint a more comprehensive image of 
respondent attitudes and needs. Finally, the survey helps to identify both system-level 
research needs and areas where local researchers need RP Unit effort. 

The RP Unit will need to decide about the effect of the current state budget crisis 
on the feasibility of planning a research agenda on data collected before the crisis gained 
so much salience. From one viewpoint, the budget crisis, despite its unpredictable 
evolution, may make the collected data totally obsolete. Respondents could offer a very 
different set of responses, especially in terms of research needs, if they knew that their 
campus would suffer a major budget reduction. From another viewpoint, the collected 
data represent real research needs that could affect improvement efforts throughout the 
state, despite the paucity of funds for such work. In this latter view, a strategic planner 
may argue that the needs exist; it is up to decision makers to find the resources to respond 
to these needs. This touches upon the very concept of strategic plans. Are strategic plans 
the foundation for pragmatic operational plans or are they idealistic “wish lists” that 
function as guides for what entities should do (but may not be able to do)? 

Another consideration will be the input of the many other stakeholders that the RP 
Unit has. Other external parties may have vastly different opinions about the RP Unit’s 
role and research agenda. Finally, the internal parties will doubtless have a pivotal 
influence on the use of these survey results. Reconciling the interests of these different 
stakeholders will determine the final shape of the Unit’s research agenda. In any case, 
more work is needed before a coherent and adaptive strategic plan — and a concomitant 
research agenda — will materialize. 
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Appendix: Copy of the Questionnaire 



From: rspsurvey [mailto:rspsurvey@ccl.cccco.edu] 

Sent: Wednesday, August 28, 2002 12:39 PM 
To: 'Research and Planning' 

Subject: Research for Strategic Planning 

2002 Survey for Strategic Planning in the Research & Planning Unit of the Chancellor's 
Office, California Community Colleges 

In order to respond to the substantial needs of our stakeholders, the 
Research & Planning Unit of the Chancellor's Office (CCC) has begun to develop its own 
strategic plan. Because you are a member of the Research & Planning Group of 
California, you are a valuable stakeholder in our unit's work. Therefore, your opinion, 
whether positive or negative, will have much value and influence in our unit's planning 
effort. Please take a few minutes to complete our survey and allow us to thank you in 
advance for your input. 

Your responses to this survey are confidential. We plan to begin the 
summary analysis of the surveys on September 13, 2002, and we hope to receive all 
survey responses before then. For your convenience, you may either e-mail this 
document to us at rspsurvey@cccco.edu, or you may send it to Shuqin Guo via regular 
mail at Chancellor's Office, Research & Planning Unit, 1102 Q Street, Sacramento, CA 
95814-6511. If you have any questions, please contact Shuqin Guo at (916) 322-5229 or 
at sguo@cccco.edu . 

For each item below, simply move your cursor to the answer space and 
type in your response. For the rating dimensions, the space to the right of each item is the 
answer space (some mail systems may not allow the table format and the answer boxes to 
appear on the screen). When you are ready to send in this completed survey, just type in 
the address of rspsurvey@cccco.edu for the destination of your e-mail and send it. 



1 . Using your impressions of the Research & Planning Unit, please rate us on the 
following dimensions, using a 0 - 9 scale, with the highest possible rating as 9, and the 
lowest possible rating as 0 (or use "DK" for Don't Know): 

Rating Dimension Your Rating 

a. credibility (objectivity): 

b. responsiveness (timeliness & completeness): 

c. analytical capability (technical competence): 

d. clarity of commimication: 

e. relevance to your research & planning: 
f your overall satisfaction with the unit: 
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Appendix: Copy of the Questionnaire (continued) 

2. Please comment about your ratings: 

3. Which topics concerning the California community college system will have the 
greatest need for research, considering a time frame of the next five years? 

4. Which research issues at your college (or district if you directly work for a district) 
would benefit the most from work at the Research & Planning Unit of the Chancellor's 
Office? 

That is the last question. Thanks! 
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